Multimedia data processing method and apparatus, storage medium, and electronic device

ABSTRACT

A multimedia data processing method is provided. The multimedia data processing method includes: establishing, by a signaling processing module deployed on a remote node, a session between a terminal device and a media processing module, and controlling the session; and receiving, by the media processing module deployed on an edge computing node after the session is established with the terminal device, multimedia data transmitted by the terminal device, and processing the multimedia data.

RELATED APPLICATION(S)

This application is a continuation application of PCT Patent ApplicationNo. PCT/CN2020/092750 filed on May 28, 2020, which claims priority toChinese Patent Application No. 201910578317.6, entitled “MULTIMEDIA DATAPROCESSING METHOD AND APPARATUS, STORAGE MEDIUM, AND ELECTRONIC DEVICE”and filed with the National Intellectual Property Administration, PRC onJun. 28, 2019, all of which are incorporated herein by reference inentirety.

FIELD OF THE TECHNOLOGY

The present disclosure relates to the field of communicationtechnologies, and more particularly, to a multimedia data processingtechnology.

BACKGROUND

Edge computing is a distributed computing architecture. In thisarchitecture, for computing such as implementation processing and dataanalysis, a large-scale service that is processed by a remote node maybe decomposed into smaller and more easily manageable parts, and theparts are distributed to edge computing nodes for processing. Becausethe edge computing nodes are closer to user terminal devices, dataprocessing and transmission can be accelerated, thereby reducinglatency.

In the related art, a service server is usually deployed on an edgecomputing node, to implement a high-quality and low-latency service.However, certain existing edge computing-based multimedia dataprocessing methods generally have problems such as high deploymentcomplexity and low system operation and maintenance efficiency.

Information partially disclosed in the background part is only used forenhancing the understanding of the background of the present disclosure,and therefore, may include information that does not constitute alimitation on the related art known to a person of ordinary skill in theart.

SUMMARY

Embodiments of the present disclosure provide a multimedia dataprocessing method, a multimedia data processing apparatus, an electronicdevice, and a computer-readable storage medium, which can reducedeployment complexity of an edge computing system and improving systemoperation and maintenance efficiency.

In one aspect, the present disclosure provides a multimedia dataprocessing method. The multimedia data processing method is performed bya server and includes: establishing, by a signaling processing moduledeployed on a remote node, a session between a terminal device and amedia processing module, and controlling the session; and receiving, bythe media processing module deployed on an edge computing node after thesession is established between the media processing module and theterminal device, multimedia data transmitted by the terminal device, andprocessing the multimedia data.

In another aspect, the present disclosure provides a multimedia dataprocessing apparatus. The multimedia processing apparatus includes amemory storing computer program instructions, and a processor coupled tothe memory and configured to execute the computer program instructionsand perform: establishing, by a signaling processing module deployed ona remote node, a session between a terminal device and a mediaprocessing module, and controlling the session; and receiving, by themedia processing module deployed on an edge computing node after thesession is established between the media processing module and theterminal device, multimedia data transmitted by the terminal device, andprocessing the multimedia data.

In yet another aspect, the present disclosure provides a non-transitorycomputer-readable storage medium storing computer program instructionsexecutable by at least one processor to perform: establishing, by asignaling processing module deployed on a remote node, a session betweena terminal device and a media processing module, and controlling thesession; and receiving, by the media processing module deployed on anedge computing node after the session is established between the mediaprocessing module and the terminal device, multimedia data transmittedby the terminal device, and processing the multimedia data.

The exemplary embodiments of the present disclosure may have some or allof the following beneficial effects:

In the multimedia data processing method provided in the embodiments ofthe present disclosure, a media processing module is deployed on an edgecomputing node, a signaling processing module is deployed on a remotenode, the signaling processing module establishes a session between aterminal device and the media processing module, and controls thesession, and the media processing module receives multimedia datatransmitted by the terminal device, and processes the multimedia data.On one hand, the signaling processing module may only need to processcontrol signaling, and quantities of network resources and computingresources occupied by the signaling processing module are not large, sothat one signaling processing module may be simultaneously responsiblefor sessions between a plurality of media processing modules distributedon different edge computing nodes and a plurality of terminal devices.Therefore, based on the method in the embodiments of the presentdisclosure, a deployment mode applied to a multimedia data processingcan be simplified, to significantly reduce deployment complexity of theedge computing nodes; and infrastructure resources that may need to bedeployed on the edge computing nodes can further be reduced, therebyreducing deployment costs. On the other hand, because the signalingprocessing module is deployed on the remote node, installation, upgrade,and maintenance of the signaling processing module also become simplerand more convenient. In addition, there is only a stateless mediaprocessing module in the edge computing node, so that an edge computingservice deployed in the edge computing node becomes a consistent singleservice type, and management and operation and maintenance work on theedge computing node become more efficient. Therefore, based on themethod in this exemplary implementation, the operation and maintenanceefficiency of the entire system can be greatly improved, and operationand maintenance costs can be reduced.

It is to be understood that, the foregoing general descriptions and thefollowing detailed descriptions are merely for illustration andexplanation purposes and are not intended to limit the presentdisclosure.

Other aspects of the present disclosure can be understood by thoseskilled in the art in light of the description, the claims, and thedrawings of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

To facilitate a better understanding of technical solutions of certainembodiments of the present disclosure, accompanying drawings aredescribed below. The accompanying drawings are illustrative of certainembodiments of the present disclosure, and a person of ordinary skill inthe art may still derive other drawings from these accompanying drawingswithout having to exert creative efforts. When the followingdescriptions are made with reference to the accompanying drawings,unless otherwise indicated, same numbers in different accompanyingdrawings may represent same or similar elements. In addition, theaccompanying drawings are not necessarily drawn to scale.

FIG. 1 is a schematic diagram of an exemplary system architecture towhich a multimedia data processing method and apparatus according to oneor more embodiments of the present disclosure is applicable;

FIG. 2 is a schematic structural diagram of a computer system adapted toimplement an electronic device according to one or more embodiments ofthe present disclosure;

FIG. 3 is a schematic diagram of a deployment architecture of amultimedia data processing apparatus according to one or moreembodiments of the present disclosure;

FIG. 4 is a schematic diagram of a deployment architecture of amultimedia data processing apparatus according to one or moreembodiments of the present disclosure;

FIG. 5 is a schematic flowchart of a multimedia data processing methodaccording to an embodiment of the present disclosure.

FIG. 6 is a schematic flowchart of a process of establishing amultimedia session according to one or more embodiments of the presentdisclosure;

FIG. 7 is a schematic interaction flowchart of a process of establishinga multimedia session according to one or more embodiments of the presentdisclosure;

FIG. 8 is a schematic interaction flowchart of a process of establishinga multimedia session according to one or more embodiments of the presentdisclosure;

FIG. 9 is a schematic flowchart of steps of processing multimedia dataaccording to one or more embodiments of the present disclosure; and

FIG. 10 is a schematic block diagram of a multimedia data processingapparatus according to one or more embodiments of the presentdisclosure.

DETAILED DESCRIPTION

Exemplary implementations are described more comprehensively withreference to the accompanying drawings. However, the exemplaryimplementations can be implemented in various forms, and are not to beunderstood as being limited to the examples described herein. Instead,the implementations are provided to make the present disclosure morecomprehensive and fully convey the idea of the exemplary implementationsto a person skilled in the art. The described features, structures, orcharacteristics may be combined in one or more implementations in anyappropriate manner. In the following description, many specific detailsare provided to give a full understanding of the implementations of thepresent disclosure. However, it is to be appreciated by a person skilledin the art that one or more of the specific details may be omittedduring practice of the technical solutions of the present disclosure, orother methods, components, apparatus, steps, or the like may be used. Inother scenarios, well-known technical solutions are not shown ordescribed in detail to avoid obscuring aspects of the presentdisclosure.

In addition, the accompanying drawings are only schematic illustrationsof the present disclosure and are not necessarily drawn to scale. Thesame reference numbers in the drawings represent the same or similarparts, and therefore, repeated descriptions thereof are omitted. Some ofthe block diagrams shown in the accompanying drawings are functionalentities and do not necessarily correspond to physically or logicallyindependent entities. The functional entities may be implemented in theform of software, or implemented in one or more hardware modules orintegrated circuits, or implemented in different networks and/orprocessor apparatuses and/or micro-controller apparatuses.

To make objectives, technical solutions, and/or advantages of thepresent disclosure more comprehensible, certain embodiments of thepresent disclosure are further elaborated in detail with reference tothe accompanying drawings. The embodiments as described are not to beconstrued as a limitation to the present disclosure. All otherembodiments obtained by a person of ordinary skill in the art withoutcreative efforts shall fall within the protection scope of embodimentsof the present disclosure.

Throughout the description, and when applicable, “some embodiments” or“certain embodiments” describe subsets of all possible embodiments, butit may be understood that the “some embodiments” or “certainembodiments” may be the same subset or different subsets of all thepossible embodiments, and can be combined with each other withoutconflict.

In one or more embodiments of the present disclosure, the term “basedon” is employed herein interchangeably with the term “according to.”

FIG. 1 is a schematic diagram of a system architecture of animplementation environment of a multimedia data processing methodaccording to an embodiment of the present disclosure.

As shown in FIG. 1, the system architecture 100 may include one or moreof terminal devices 101, 102, and 103, a base station 104, an edgecomputing node 105, a local network computing platform 106, and anInternet computing platform 107. The edge computing node 105 may be acomputing platform close to a network access location, and may belocated in an edge data center (DC). The local network computingplatform 106 and the Internet computing platform 107 may be used asremote nodes in the present disclosure. The local network computingplatform 106 may be located in a local DC. The Internet computingplatform 107 may be located in an Internet data center (IDC). Inaddition, the system architecture is a three-layer network structure inFIG. 1, and may be alternatively a two-layer network structure inanother exemplary embodiment. That is, the edge computing node 105 maybe directly communicatively connected to the Internet computing platform107. The system architecture may be alternatively a network structure ofmore than three layers. This is not particularly limited in thisexemplary embodiment.

The terminal devices 101, 102, and 103 may be various electronicdevices, including, but not limited to, cameras, desktop computers,portable computers, smartphones, tablet computers, and the like. Each ofthe edge computing node 105, the local network computing platform 106,and the Internet computing platform 107 includes a server. It is to beunderstood that the quantity of terminal devices, the quantity ofnetworks, and the quantity of servers in FIG. 1 are merely exemplary.There may be any quantity of terminal devices, networks, and serversaccording to implementation requirements. For example, the edgecomputing node 105, the local network computing platform 106, and theInternet computing platform 107 may be a server cluster including aplurality of servers. In another example, the local network computingplatform 106 and the Internet computing platform 107 may bealternatively cloud computing platforms, thereby implementing anedge-cloud collaboration architecture. That is, the edge computing nodeand remote cloud computing platforms function collaboratively.

The multimedia data processing method provided in the embodiments of thepresent disclosure is generally performed by a server, andcorrespondingly, a multimedia data processing apparatus is generallydisposed in the server. For example, in an exemplary embodiment, amobile edge computing (MEC) application may be deployed on the edgecomputing node 105, to implement quick downloading of various content,services, and applications in a network, to provide consumers withuninterrupted high-quality network experience. Network traffic that isnot processed by the MEC application may be forwarded to a UPF entity inthe local network computing platform 106 by a UPF entity in the edgecomputing node 105, and finally enter the Internet computing platform107.

FIG. 2 is a schematic structural diagram of a computer system adapted toimplement an electronic device according to an embodiment of the presentdisclosure.

A computer system 200 of the electronic device shown in FIG. 2 is merelyan example, and does not constitute any limitation on functions and useranges of the embodiments of the present disclosure.

As shown in FIG. 2, the computer system 200 includes a centralprocessing unit (CPU) 201. The CPU may perform various appropriateactions and processing according to a program stored in a read-onlymemory (ROM) 202 or a program loaded from a storage portion 208 into arandom access memory (RAM) 203. The RAM 203 further stores variousprograms and data required for system operations. The CPU 201, the ROM202, and the RAM 203 are connected to each other by using a bus 204. Aninput/output (I/O) interface 205 is also connected to the bus 204.

The following components are connected to the I/O interface 205: aninput part 206 including a keyboard, a mouse, or the like, an outputpart 207 including a cathode ray tube (CRT), a liquid crystal display(LCD), a speaker, or the like, a storage part 208 including a hard disk,or the like, and a communication part 209 including a network interfacecard such as a local area network (LAN) card or a modem. Thecommunication part 209 performs communication processing by using anetwork such as the Internet. A driver 210 is also connected to the I/Ointerface 205 as required. A removable medium 211, such as a magneticdisk, an optical disc, a magneto-optical disk, or a semiconductormemory, is installed on the drive 210 as required, so that a computerprogram read from the removable medium is installed into the storagepart 208 as required.

Particularly, according to an embodiment of the present disclosure, theprocesses described in the following by referring to the flowcharts maybe implemented as computer software programs. For example, thisembodiment of the present disclosure includes a computer programproduct, the computer program product includes a computer programcarried on a computer-readable medium, and the computer program includesprogram code used for performing the methods shown in the flowcharts. Insuch an embodiment, the computer program may be downloaded and installedfrom a network through the communication part 209, and/or installed fromthe removable medium 211. When the computer program is executed by theCPU 201, various functions defined in the method and apparatus of thepresent disclosure are executed. In some embodiments, the computersystem 200 may further include an artificial intelligence (AI)processor. The AI processor is configured to process computingoperations related to machine learning.

The computer-readable medium shown in the present disclosure may be acomputer-readable signal medium or a computer-readable storage medium orany combination of the two. The computer-readable storage medium may be,for example, but is not limited to, an electric, magnetic, optical,electromagnetic, infrared, or semi-conductive system, apparatus, orcomponent, or any combination thereof. A more specific example of thecomputer-readable storage medium may include but is not limited to: anelectrical connection having one or more wires, a portable computermagnetic disk, a hard disk, a RAM, a ROM, an erasable programmableread-only memory (EPROM), a flash memory, an optical fiber, a compactdisk read-only memory (CD-ROM), an optical storage device, a magneticstorage device, or any appropriate combination thereof.

In the present disclosure, the computer-readable storage medium may beany tangible medium containing or storing a program, and the program maybe used by or used in combination with an instruction execution system,an apparatus, or a device. In the present disclosure, acomputer-readable signal medium may include a data signal being in abaseband or propagated as a part of a carrier wave, the data signalcarrying computer-readable program code. The propagated data signal maybe in a plurality of forms, including but not limited to, anelectromagnetic signal, an optical signal, or any appropriatecombination thereof. The computer-readable signal medium may be furtherany computer-readable medium in addition to a computer-readable storagemedium. The computer-readable medium may send, propagate, or transmit aprogram that is used by or used in conjunction with an instructionexecution system, an apparatus, or a device. The program code containedin the computer readable medium may be transmitted by using anyappropriate medium, including but not limited to: a wireless medium, awire, an optical cable, radio frequency (RF), any suitable combinationthereof, or the like.

The flowcharts and block diagrams in the accompanying drawingsillustrate possible system architectures, functions and operations thatmay be implemented by a system, a method, and a computer program productaccording to various embodiments of the present disclosure. In thisregard, each box in a flowchart or a block diagram may represent amodule, a program segment, or a part of code. The module, the programsegment, or the part of code includes one or more executableinstructions used for implementing designated logic functions. In someimplementations used as substitutes, functions annotated in boxes mayalternatively occur in a sequence different from that annotated in anaccompanying drawing. For example, actually two boxes shown insuccession may be performed basically in parallel, and sometimes the twoboxes may be performed in a reverse sequence. This is determined by arelated function. Each box in a block diagram and/or a flowchart and acombination of boxes in the block diagram and/or the flowchart may beimplemented by using a dedicated hardware-based system configured toperform a specified function or operation, or may be implemented byusing a combination of dedicated hardware and a computer instruction.

The involved units described in the embodiments of the presentdisclosure may be implemented in a software manner, or may beimplemented in a hardware manner, and the described units may also bedisposed in a processor. Names of the units do not constitute alimitation on the units in certain particular implementations.

According to another aspect, the present disclosure further provides acomputer-readable medium. The computer-readable medium may be includedin the electronic device described in the foregoing embodiments, or mayexist alone and is not disposed in the electronic device. Thecomputer-readable medium carries one or more programs, the one or moreprograms, when executed by the electronic device, causing the electronicdevice to implement the method described in the following embodiments.

The technical solutions in the embodiments of the present disclosure aredescribed below in detail.

Based on an edge computing network deployment architecture shown in FIG.1, there may be different solutions for different implementationscenarios.

A video surveillance implementation scenario based on edge computing isused as an example. An implementation is shown in FIG. 3. In thisimplementation solution, a signaling processing module and a mediaprocessing module are deployed on an edge computing node. The signalingprocessing module controls access of a video. The media processingmodule performs video analysis based on a graphics processing unit (GPU)and a deep neural network on the received video, to output structuredvideo data, and then transmits the structured video data to a dataprocessing module deployed on a remote node. The data processing moduleperforms processing such as retrieval and analysis based on thestructured video data.

Based on this implementation solution, on one hand, the edge computingnode deployed on a user side performs video session signaling processingand media processing, thereby improving video real-time processingefficiency. On the other hand, the data processing module deployed onthe remote node receives and processes the structured video datatransmitted by the edge computing node, which can avoid transmission oforiginal video media data between the edge computing node and the remotenode, thereby effectively decreasing a requirement for network bandwidthbetween the edge computing node and the remote node.

A video on-demand implementation scenario based on edge computing isused as an example. An implementation is: deploying a signalingprocessing module and a media processing module on an edge computingnode including a stream media server, and deploying a content deliverynetwork (CDN) management server on a remote node. When a user requests astream media on-demand service, the request is first submitted to theCDN management server deployed on the remote node. The CDN managementserver selects, according to a corresponding policy, a stream mediaserver deployed in the corresponding edge computing node, and forwardsthe request of the user to the stream media server, to provide the userwith a nearest high-quality and low-latency video on-demand service. Inaddition, video data transmission between a user terminal device and theremote node is reduced.

In the implementation solutions of the two foregoing scenarios, aservice server may need to be deployed on each edge computing node, andthe service server includes a signaling processing module and a mediaprocessing module. The signaling processing module and the mediaprocessing module are simultaneously deployed on the edge computingnode, which increases system deployment complexity, and increases systemdeployment costs. In addition, during system upgrade or expansion, thesignaling processing module and the media processing module that aredeployed in each edge computing node may need to be separately upgradedor expanded, and system operation and maintenance efficiency and costsare relatively high.

In a multimedia session service system, the signaling processing moduleis responsible for establishing a media session. Throughout one sessionfrom initiation to end, the entire signaling interaction occupies veryfew computing resources, and compared with network resources occupied bymedia transmission, network resources occupied by the entire signalinginteraction may be omitted. Based on the above, the present disclosureprovides a multimedia data processing method. The multimedia dataprocessing method may be applicable to a multimedia sessionimplementation-oriented deployment architecture shown in FIG. 4.

Referring to FIG. 4, in the deployment architecture, a multimedia dataprocessing apparatus includes a signaling processing module and mediaprocessing modules. The signaling processing module is deployed on aremote node, for example, the foregoing local network computing platform106 or Internet computing platform 107. The media processing module isdeployed on an edge computing node 105, to implement a simple andflexible edge computing implementation. Based on the deploymentarchitecture, signaling processing of multimedia sessions between userterminal devices and the multimedia data processing apparatus isperformed by the signaling processing module deployed on the remotenode, and multimedia processing between the user terminal devices andthe multimedia data processing apparatus is performed by the mediaprocessing modules deployed on edge computing nodes in a scatteredmanner.

The multimedia data processing method based on the foregoing deploymentarchitecture is described below with reference to FIG. 5. Referring toFIG. 5, the method may include the following step S510 and step S520.

Step S510: A signaling processing module deployed on a remote nodeestablishes a session between a terminal device and a media processingmodule, and controls the session.

In this exemplary implementation, the session is a process in which aterminal device communicates with a media processing module. Thesignaling processing module may establish a session between the terminaldevice and the media processing module. In addition, the signalingprocessing module may further control the session to be kept and end thesession.

The signaling processing module is generally deployed on a remote room.A video surveillance implementation scenario is used as an example.Protocols supported by the signaling processing module may includemultimedia session protocols such as a session initiation protocol (SIP)and a real-time stream protocol (RTSP). According to differentimplementation scenarios and requirements, protocols supported by thesignaling processing module may alternatively include other multimediasession protocols such as an HTTP Live Streaming (HLS) and a real timemessaging protocol (RTMP). In a non-multimedia session implementationscenario, the signaling processing module may further support anon-multimedia session protocol. This exemplary embodiment is notlimited thereto.

Referring to FIG. 6, in this exemplary implementation, the sessionbetween the terminal device and the media processing module may beestablished through the following step S610 to step S640. Details are asfollows:

Step S610: Initiate a call request including information about a datareception address to the terminal device.

For example, after receiving a camera video access request initiated bya service administrator, the signaling processing module may initiate,for example, a SIP\RTSP call request to a camera. The call request mayinclude, for example, information about a data reception address in asession description protocol (SDP) format. The information about thedata reception address may include information such as an IP address ofthe media processing module and a connection port number.

Step S620: Receive response information of the terminal device for thecall request, the response information including information about adata transmission address.

For example, after receiving the call request of the signalingprocessing module, the camera may return response information to thesignaling processing module. The response information may include, forexample, information about a data transmission address in an SDP format.The information about the data transmission address may includeinformation such as a camera identifier (for example, an ID) and asource IP address and a transport port number that are used fortransmitting multimedia data by the camera.

Step S630: Transmit the information about the data transmission addressto the media processing module, and instruct the media processing moduleto receive the multimedia data according to the data reception address.

Step S640: Transmit session establishment information to the terminaldevice.

During the session, a UPF entity may directly exchange protocols of acontrol plane such as a SIP or an RTSP between the terminal device andthe signaling processing module according to a configuration policy. Inthis exemplary implementation, the UPF entity may be a user plane devicein a 5G network, but is not limited thereto. For example, the UPF entitymay be alternatively a non-mobile communication gateway device such as awireless access gateway (for example, a Wi-Fi gateway) or a wired accessgateway. Forwarding policies of the UPF entity for multimedia data mayinclude a static policy and a dynamic policy. Details are as follows:

Based on the static policy, a forwarding policy may need to bepre-configured in a UPF entity deployed on the edge computing node, theforwarding policy including: forwarding multimedia data whose receivingport is within a preset range to the media processing module, theinformation about the data reception address including a targetreceiving port within the preset range.

In addition, during multimedia session negotiation, it is necessary torequire that the terminal device set a receiving port in the presetrange. The foregoing video surveillance implementation scenario is usedas an example. Based on a SIP, during SDP negotiation, the camera mayneed to limit the target receiving port of the transmitted multimediadata to 10000 to 20000, and a forwarding policy is then configured inthe UPF entity. Any received IP packet, for example, a user datagramprotocol (UDP) packet, that is of a real-time transport protocol (RTP)type, whose target receiving port is in a range of 10000 to 20000, andthat is transmitted by the camera is forwarded to a port correspondingto the media processing module.

FIG. 7 is a schematic diagram of a process of a session based on astatic policy. The process includes step S710 to step S760. Details areas follows:

Step S710: After receiving a camera video access request initiated by aservice administrator, the signaling processing module initiates aSIP\RTSP call request to a camera. The call request may include, forexample, information about a data reception address in an SDP format.The information about the data reception address may need to limit thata data receiving port is a free port A within a range of 10000 to 20000.A connection address is specified as an IP address of the mediaprocessing module.

Step S720: After receiving the call request of the signaling processingmodule, the camera may return response information to the signalingprocessing module. The response information may include, for example,information about a data transmission address in an SDP format. Theinformation about the data transmission address may include informationsuch as a camera identifier (for example, an ID) and a source IP addressand a transport port number that are used for transmitting multimediadata by the camera.

Step S730: After receiving the response information of the camera, thesignaling processing module transmits an instruction to the mediaprocessing module, the instruction including information such as thecamera identifier and the source IP address and the transport portnumber that are used for transmitting the multimedia data by the camera,to instruct the media processing module to prepare to start to receive,at the port A, the multimedia data transmitted by the camera.

Step S740: The signaling processing module transmits sessionestablishment information to the terminal device, to confirm that thesession has been established.

Step S750: The camera starts to transmit a multimedia data packet to theUPF entity of the edge computing node, a target receiving port of themultimedia data packet being the port A.

Step S760: After receiving the multimedia data packet transmitted by thecamera, the UPF entity of the edge computing node forwards themultimedia data packet to the port A of the media processing moduleaccording to a preset forwarding policy.

Based on the dynamic policy, the information about the data receptionaddress includes a target receiving port; and before the step oftransmitting the information about the data transmission address to themedia processing module, the method further includes: configuring aforwarding policy in a UPF entity deployed on the edge computing node,the forwarding policy including: forwarding multimedia data whosereceiving port is the target receiving port to the media processingmodule.

The foregoing video surveillance implementation scenario is used as anexample. For example, based on a SIP, a free port B may be randomlyselected as a target receiving port during SDP negotiation, and aforwarding policy is then configured in the UPF entity. Any received IPpacket, for example, a UDP packet, that is of an RTP type, whose targetreceiving port is the port B, and that is transmitted by the camera isforwarded to the port B corresponding to the media processing module.

FIG. 8 is a schematic diagram of a process of a session based on adynamic policy. The process includes step S810 to step S870. Details areas follows:

Step S810: After receiving a camera video access request initiated by aservice administrator, the signaling processing module initiates aSIP\RTSP call request to a camera. The call request may include, forexample, information about a data reception address in an SDP format.The information about the data reception address may need to limit thata data receiving port is a randomly selected free port B. A connectionaddress is specified as an IP address of the media processing module.

Step S820: After receiving the call request of the signaling processingmodule, the camera may return response information to the signalingprocessing module. The response information may include, for example,information about a data transmission address in an SDP format. Theinformation about the data transmission address may include informationsuch as a camera identifier (for example, an ID), and a source IPaddress and a transport port number that are used for transmittingmultimedia data by the camera.

Step S830: After receiving the response information of the camera, thesignaling processing module transmits an instruction to the UPF entity,to set a forwarding policy of the UPF entity. The forwarding policyrequires that any received IP packet, for example, a UDP packet, that isof an RTP type, whose target receiving port is the port B, and that istransmitted by the camera is forwarded to the port B corresponding tothe media processing module.

Step S840: The signaling processing module transmits an instruction tothe media processing module, the instruction including information suchas the camera identifier and the source IP address and the transportport number that are used for transmitting the multimedia data by thecamera, to instruct the media processing module to prepare to start toreceive, at the port B, the multimedia data transmitted by the camera.

Step S850: The signaling processing module transmits sessionestablishment information to the terminal device, to confirm that thesession has been established.

Step S860: The camera starts to transmit a multimedia data packet to theUPF entity of the edge computing node, a target receiving port of themultimedia data packet being the port B.

Step S870: After receiving the multimedia data packet transmitted by thecamera, the UPF entity of the edge computing node forwards themultimedia data packet to the port B of the media processing moduleaccording to a preset forwarding policy.

In addition, after the session is ended, the foregoing set dynamicforwarding policy may be deleted, to avoid the impact on establishmentof a subsequent session.

Step S520: The media processing module deployed on an edge computingnode receives, after the session is established between the mediaprocessing module and the terminal device, multimedia data transmittedby the terminal device, and processes the multimedia data.

In this exemplary implementation, the media processing module isdeployed on the edge computing node. The media processing module may beresponsible for processing of multimedia data, including reception of anaudio-video media packet, decoding of audio-video data, audio-videocontent analysis based on a deep learning method, and the like. Ifaudio-video content analysis is performed by using a deep learningmethod, the media processing module usually may need a GPU to improvecomputing performance in media analysis. Based on this, in thisexemplary implementation, the processing the multimedia data may includestep S910 and step S920.

Step S910: Decode the multimedia data to obtain decoded data.

In this exemplary implementation, lossless decoding may be performed onthe multimedia data. That is, a decoded image is strictly the same as anoriginal image, and compression is restorable or unbiased withoutdistortion. To reduce a data volume, lossy decoding may be alternativelyused. A decoding module may be a software module or a hardware module.This is not particularly limited in this exemplary embodiment.

Step S920: Perform computational analysis on the decoded data to convertthe decoded data into structured multimedia data.

In this exemplary implementation, structural extraction may be performedon important information in content of the multimedia data, andprocessing means such as spatiotemporal segmentation, featureextraction, object recognition, and deep learning may be used toorganize the important information into text information or visualizedgraphical information that can be understood by a computer or a person.Through structuring of the multimedia data, a large quantity of originalsurveillance videos that have slow response and of which a majority hasno implementation value can be finely compressed, thereby greatlyimproving the search and investigation efficiency, and greatly reducingoccupied storage space.

Referring to step S770 in FIG. 7 or step S880 in FIG. 8: Afterconverting the multimedia data into structured multimedia data, themedia processing module may further transmit the structured multimediadata to the remote node. For example, the media processing moduletransmits the structured multimedia data to the data processing moduleof the local network computing platform 106 or the Internet computingplatform 107.

In this exemplary implementation, the data processing module mayperform, according to requirements of products and by using thestructured multimedia data, operations such as multimedia retrieval orrecognition, for example, face retrieval, face recognition, and humanbody behavior analysis. For example, structured databases such as a facephoto database, a face feature database, a behavior picture and featurelibrary, and a vehicle image and feature library may be establishedbased on processing structures, and a corresponding retrieval engine isestablished by combining the databases and associated video clipdatabases, to mine depth information of various databases and make fulluse of functions of big data, thereby improving implementation value ofvideo data, and improving analysis and prediction functions for thevideo data.

In the multimedia data processing method in this exemplaryimplementation, the media processing module is deployed on the edgecomputing node, and the signaling processing module is deployed on theremote node. A quantity of deployed media processing modules depends ona quantity of accessed terminal devices, for example, a quantity ofcameras. The signaling processing module only may need to processcontrol signaling, and quantities of network resources and computingresources occupied by the signaling processing module are not large, sothat one signaling processing module may be simultaneously responsiblefor sessions between a plurality of media processing modules distributedon different edge computing nodes and a plurality of terminal devices.For example, a SIP signaling processing module deployed on ageneral-purpose server having an X86 architecture can generallysimultaneously support thousands of SIP calls, that is, cansimultaneously process thousands of sessions between cameras and mediaprocessing modules that support a SIP. For example, the signalingprocessing module starts one or more service instances, and servicerequirements of media processing modules of all edge computing nodes andterminal devices in the whole network can be met. Therefore, based onthe method in this exemplary implementation, a deployment mode appliedto a multimedia data processing can be simplified, to significantlyreduce deployment complexity of the edge computing nodes; andinfrastructure resources required by the edge computing nodes canfurther be reduced, thereby reducing deployment costs.

In addition, because the signaling processing module is deployed on theremote node, installation, upgrade, and maintenance of the signalingprocessing module also become simpler and more convenient. For the edgecomputing node, the signaling processing module is a stateful computingservice, and the media processing module is a stateless computingservice. After the stateful signaling processing module is deployed onthe remote node, there is only a stateless media processing module inthe edge computing node, so that an edge computing service deployed inthe edge computing node becomes a consistent single service type, andmanagement and operation and maintenance work on the edge computing nodebecome more efficient. Therefore, based on the method in this exemplaryimplementation, the operation and maintenance efficiency of the entiresystem can be greatly improved, and operation and maintenance costs canbe reduced.

In addition, in the multimedia data processing method in this exemplaryimplementation, for the UPF entity, because it is unnecessary to offloadsignaling control traffic to the edge computing node, forwarding policyconfiguration of the UPF entity also becomes simpler and moreconvenient. Therefore, complexity of a traffic offloading function ofthe entire system can further be reduced.

Descriptions are made in the foregoing exemplary embodiments by usingthe video surveillance implementation scenario as an example. In otherexemplary embodiments of the present disclosure, the multimedia dataprocessing method in the present disclosure may be alternativelyapplicable to other implementation scenarios such as voice monitoring,an Internet of vehicles and automatic driving implementation, videoon-demand, and game entertainment, which also fall within the protectionscope of the present disclosure.

Although the steps of the method in the present disclosure are describedin a specific order in the accompanying drawings, this does not requireor imply that the steps are bound to be performed in the specific order,or all the steps shown are bound to be performed to achieve an expectedresult. Additionally or alternatively, some steps may be omitted, aplurality of steps may be combined into one step for execution, and/orone step may be decomposed into a plurality of steps for execution, andthe like.

Further, in this exemplary implementation, a multimedia data processingapparatus is further provided. Referring to FIG. 10, the multimedia dataprocessing apparatus 1000 may include a signaling processing module 1010and a media processing module 1020.

The signaling processing module 1010 is deployed on a remote node, andis configured to: establish a session between a terminal device and themedia processing module 1020, and control the session.

The media processing module 1020 is deployed on an edge computing node,and is configured to: receive, after the session is established betweenthe media processing module and the terminal device, multimedia datatransmitted by the terminal device, and process the multimedia data.

In this exemplary implementation, the apparatus includes a plurality ofsignaling processing modules 1010 and a plurality of media processingmodules 1020, a quantity of the signaling processing modules 1010 beingless than a quantity of the media processing modules 1020.

In this exemplary implementation, the apparatus includes one signalingprocessing module 1010 and a plurality of media processing modules 1020.

In this exemplary implementation, the remote node includes a localnetwork computing platform and/or an Internet computing platform.

In this exemplary implementation, the signaling processing module 1010establishes the session between the terminal device and the mediaprocessing module 1020 through the following steps: initiating a callrequest including information about a data reception address to theterminal device; receiving response information of the terminal devicefor the call request, the response information including informationabout a data transmission address; transmitting the information aboutthe data transmission address to the media processing module 1020, andinstructing the media processing module 1020 to receive the multimediadata according to the data reception address; and transmitting sessionestablishment information to the terminal device.

In this exemplary implementation, the signaling processing module 1010is further configured to: pre-configure a forwarding policy in a UPFentity deployed on the edge computing node, the forwarding policyincluding: forwarding multimedia data whose receiving port is within apreset range to the media processing module 1020, the information aboutthe data reception address including a target receiving port within thepreset range.

In this exemplary implementation, the information about the datareception address includes a target receiving port; and beforeperforming the step of transmitting the information about the datatransmission address to the media processing module 1020, the signalingprocessing module 1010 is further configured to: configure a forwardingpolicy in a UPF entity deployed on the edge computing node, theforwarding policy including: forwarding multimedia data whose receivingport is the target receiving port to the media processing module 1020.

In this exemplary implementation, the signaling processing module 1010is further configured to: delete the forwarding policy after the sessionis ended.

In this exemplary implementation, the apparatus includes: a UPF entity,configured to forward, according to the forwarding policy, themultimedia data transmitted by the terminal device to the mediaprocessing module 1020.

In this exemplary implementation, the controlling the session includes:keeping the session and ending the session.

In this exemplary implementation, the media processing module 1020processes the multimedia data through the following steps: decoding themultimedia data to obtain decoded data; and performing computationalanalysis on the decoded data to convert the decoded data into structuredmultimedia data.

In this exemplary implementation, the media processing module 1020 isfurther configured to: transmit the structured multimedia data to theremote node.

The term unit (and other similar terms such as subunit, module,submodule, etc.) in this disclosure may refer to a software unit, ahardware unit, or a combination thereof. A software unit (e.g., computerprogram) may be developed using a computer programming language. Ahardware unit may be implemented using processing circuitry and/ormemory. Each unit can be implemented using one or more processors (orprocessors and memory). Likewise, a processor (or processors and memory)can be used to implement one or more units. Moreover, each unit can bepart of an overall unit that includes the functionalities of the unit.

Details of the modules or units of the multimedia data processingapparatus have been described in the corresponding multimedia dataprocessing method. Therefore, details are not described herein again.

Although a plurality of modules or units of a device configured toperform actions are discussed in the foregoing detailed description,such division is not mandatory. Actually, according to theimplementations of the present disclosure, the features and functions oftwo or more modules or units described above may be implemented in onemodule or unit. On the contrary, the features and functions of onemodule or unit described above may be further divided to be embodied bya plurality of modules or units.

Other implementations of the present disclosure are apparent to a personskilled in the art from consideration of the present disclosure andpractice of the present disclosure. The present disclosure is intendedto cover any variations, uses, or adaptive changes of the presentdisclosure. These variations, uses, or adaptive changes follow thegeneral principles of the present disclosure and include common generalknowledge or common technical means in the art, which are not disclosedin the present disclosure. The present disclosure and the embodimentsare considered as merely exemplary, and the scope and spirit of thepresent disclosure are pointed out in the following claims.

It is to be understood that the present disclosure is not limited to theprecise structures described above and shown in the accompanyingdrawings, and various modifications and changes can be made withoutdeparting from the scope of the present disclosure. The scope of thepresent disclosure is subject only to the appended claims.

What is claimed is:
 1. A multimedia data processing method, performed bya server, the method comprising: establishing, by a signaling processingmodule deployed on a remote node, a session between a terminal deviceand a media processing module, and controlling the session; andreceiving, by the media processing module deployed on an edge computingnode after the session is established between the media processingmodule and the terminal device, multimedia data transmitted by theterminal device, and processing the multimedia data.
 2. The multimediadata processing method according to claim 1, wherein establishing thesession comprises: initiating a call request including information abouta data reception address to the terminal device; receiving responseinformation of the terminal device for the call request, the responseinformation comprising information about a data transmission address;transmitting the information about the data transmission address to themedia processing module, and instructing the media processing module toreceive the multimedia data according to the data reception address; andtransmitting session establishment information to the terminal device.3. The multimedia data processing method according to claim 2, furthercomprising: pre-configuring a forwarding policy in a user plane function(UPF) entity deployed on the edge computing node, the forwarding policyincluding: forwarding multimedia data whose receiving port is within apreset range to the media processing module, the information about thedata reception address including a target receiving port within thepreset range.
 4. The multimedia data processing method according toclaim 2, wherein the information about the data reception addressincludes a target receiving port; and the method further comprises:configuring a forwarding policy in a UPF entity deployed on the edgecomputing node, the forwarding policy including: forwarding multimediadata whose receiving port is the target receiving port to the mediaprocessing module.
 5. The multimedia data processing method according toclaim 4, further comprising: deleting the forwarding policy after thesession is ended.
 6. The multimedia data processing method according toclaim 3, further comprising: forwarding, by the UPF entity according tothe forwarding policy, the multimedia data transmitted by the terminaldevice to the media processing module.
 7. The multimedia data processingmethod according to claim 1, wherein controlling the session comprises:keeping the session and ending the session.
 8. The multimedia dataprocessing method according to claim 1, wherein processing themultimedia data comprises: decoding the multimedia data to obtaindecoded data; and performing computational analysis on the decoded datato convert the decoded data into structured multimedia data.
 9. Themultimedia data processing method according to claim 8, furthercomprising: transmitting the structured multimedia data to the remotenode.
 10. A multimedia data processing apparatus, comprising: a memorystoring computer program instructions; and a processor coupled to thememory and configured to execute the computer program instructions andperform: establishing, by a signaling processing module deployed on aremote node, a session between a terminal device and a media processingmodule, and controlling the session; and receiving, by the mediaprocessing module deployed on an edge computing node after the sessionis established between the media processing module and the terminaldevice, multimedia data transmitted by the terminal device, andprocessing the multimedia data.
 11. The multimedia data processingapparatus according to claim 10, wherein the processor is furtherconfigured to execute the computer program instructions and perform:initiating a call request including information about a data receptionaddress to the terminal device; receiving response information of theterminal device for the call request, the response informationcomprising information about a data transmission address; transmittingthe information about the data transmission address to the mediaprocessing module, and instructing the media processing module toreceive the multimedia data according to the data reception address; andtransmitting session establishment information to the terminal device.12. The multimedia data processing apparatus according to claim 11,wherein the processor is further configured to execute the computerprogram instructions and perform: pre-configuring a forwarding policy ina user plane function (UPF) entity deployed on the edge computing node,the forwarding policy including: forwarding multimedia data whosereceiving port is within a preset range to the media processing module,the information about the data reception address including a targetreceiving port within the preset range.
 13. The multimedia dataprocessing apparatus according to claim 11, wherein the informationabout the data reception address includes a target receiving port, andwherein the processor is further configured to execute the computerprogram instructions and perform: configuring a forwarding policy in aUPF entity deployed on the edge computing node, the forwarding policyincluding: forwarding multimedia data whose receiving port is the targetreceiving port to the media processing module.
 14. The multimedia dataprocessing apparatus according to claim 13, wherein the processor isfurther configured to execute the computer program instructions andperform: deleting the forwarding policy after the session is ended. 15.The multimedia data processing apparatus according to claim 12, whereinthe processor is further configured to execute the computer programinstructions and perform: forwarding, by the UPF entity according to theforwarding policy, the multimedia data transmitted by the terminaldevice to the media processing module.
 16. The multimedia dataprocessing apparatus according to claim 10, wherein controlling thesession comprises: keeping the session and ending the session.
 17. Themultimedia data processing apparatus according to claim 10, whereinprocessing the multimedia data comprises: decoding the multimedia datato obtain decoded data; and performing computational analysis on thedecoded data to convert the decoded data into structured multimediadata.
 18. The multimedia data processing apparatus according to claim17, wherein the processor is further configured to execute the computerprogram instructions and perform: transmitting the structured multimediadata to the remote node.
 19. A non-transitory computer-readable storagemedium storing computer program instructions executable by at least oneprocessor to perform: establishing, by a signaling processing moduledeployed on a remote node, a session between a terminal device and amedia processing module, and controlling the session; and receiving, bythe media processing module deployed on an edge computing node after thesession is established between the media processing module and theterminal device, multimedia data transmitted by the terminal device, andprocessing the multimedia data.
 20. The non-transitory computer-readablestorage medium according to claim 19, wherein the computer programinstructions are executable by the at least one processor to furtherperform: initiating a call request including information about a datareception address to the terminal device; receiving response informationof the terminal device for the call request, the response informationcomprising information about a data transmission address; transmittingthe information about the data transmission address to the mediaprocessing module, and instructing the media processing module toreceive the multimedia data according to the data reception address; andtransmitting session establishment information to the terminal device.